Data Ingestion
Collecting data from various sources (APIs, DBs, files, etc.)
Loading
Structured. Scalable. Ready to scale.
Collecting data from various sources (APIs, DBs, files, etc.)
Extract, Transform, Load/Load, Transform operations
Scheduling, monitoring workflows (Airflow, Luigi, etc.)
Stream processing using Kafka, Flink, Spark Streaming
Building centralized data repositories (e.g., S3 + Glue)
Designing & managing DWHs (Snowflake, Redshift, BigQuery)
Combining data from different systems
Periodic data processing (daily, hourly, etc.)
Moving data between systems or to the cloud